Horticultural monitoring system

ABSTRACT

An example plant monitoring system can include sensors connected to a horticultural monitor. The sensors can measure the soil, the environment, individual plant characteristics, etc. At least one horticultural monitor can send sensor data to a local beacon. The local beacon can then receive the data and send it to cloud resources which can process the data. The cloud resources can determine the health of the plant and monitor its development. A user can track this health and development using an interaction device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and the benefit of U.S. ProvisionalPatent Application No. 62/318,110, entitled “PLANT MONITORING SYSTEM”and filed on Apr. 4, 2016, the contents of which are hereby incorporatedby reference in their entirety.

FIELD OF THE INVENTION

This disclosure relates to the cultivation and monitoring of plants.

BACKGROUND

Traditionally, growing plants and gardening has required closesupervision of the plants and expert knowledge. For this reason, manypeople either lack the time to try gardening or get frustrated by theirlack of knowledge. Furthermore, when gardeners are away (e.g., onvacation), they are sometimes required to ask neighbors to care fortheir plants. What is needed is an improved system for the monitoringand care of plants.

SUMMARY

Additional features and advantages of the disclosure will be set forthin the description which follows, and in part will be obvious from thedescription, or can be learned by practice of the herein disclosedprinciples. The features and advantages of the disclosure can berealized and obtained by means of the instruments and combinationsparticularly pointed out in the appended claims. These and otherfeatures of the disclosure will become more fully apparent from thefollowing description and appended claims, or can be learned by thepractice of the principles set forth herein.

An example plant monitoring system can include sensors connected to ahorticultural monitor. The sensors can measure one or more of the soil,the environment, individual plant characteristics, water level,humidity, sunlight, temperature, wind speed and direction, time,seasonal data, etc. A monitor (e.g., a photo or video monitor) can alsobe included as part of the plan monitoring system. At least onehorticultural monitor can send sensor data to a local beacon. The localbeacon can then receive the data and send it to cloud resources whichcan process the data. The cloud resources can determine the health ofthe plant and monitor its development based at least in part on the datareceived. A user can track the health and development of plants using aninteraction device and can also track progress and can control at leastsome components remotely.

In a first exemplary embodiment, there is provided acomputer-implemented method. The method includes capturing data for amonitored plant via one or more sensors, comparing, via a processor, thedata for the monitored plant with data associated with other plants of asame species as the monitored plant and at one or more plant life stagesto yield a comparison, and determining, based on the comparison and viaa life stage determination algorithm, a one of the plant life stagesassociated with the monitored plant.

The method can include, when the determining indicates that the one ofthe plant life stages for the monitored plant is a target plant lifestage, sending a push notification to a user device, the pushnotification indicating that the plant has reached the target plant lifestage. The method can also include, when the determining indicates thatthe one of the plant life stages for the monitored plant is a targetplant life stage, sending a push notification to a harvester device, thepush notification comprising instructions for the harvester device toharvest at least a portion of the monitored plant.

In the method, the data for the monitored plant can include image data.The method can then include determining at least one of a species of themonitored plant and the one of the plant life stages for the monitoredplant based on the image data.

When the monitored plant is a fruit bearing plant, the method caninclude estimating, based on the data for the monitored plant, aquantity of a type of fruit that is available for harvesting from theplant.

The method can include transmitting, over a wireless interface, the datafor the monitored plant from a horticultural monitor to a local beacon,and transmitting the data for the monitored plant from the local beaconto cloud resources via a wide area network, wherein the comparing isperformed by the cloud resources.

In the method, at least one of the plant life stages comprises a plantlife stage associated with a suboptimal health condition. The method canthen include, when the determining indicates that the one of the plantlife stages for the monitored plant is associated with the suboptimalhealth condition, sending a push notification to a user device, the pushnotification indicating that a health of the monitored plant issuboptimal.

The method can include determining, using the data and a healthdetermination algorithm, a cause for the health of the monitor plant tobe suboptimal. The method can then include, based on the cause,providing one or more instructions to the user device for enhancing thehealth of the monitored plant. The method can also include, based on thecause, determining a remedial measure for enhancing the health of themonitored plant that does not require human intervention andimplementing the remedial measure. The remedial measure can includeactivating at least one of an irrigation system or a nutrient deliverysystem.

In the method, the sensors include at least one of a soil sensor, acamera, or a water sensor.

In a second exemplary embodiment, a system is provided that includes aprocessing system having at least one processor and a computer-readablemedium and one or more horticultural monitors communicatively coupled tothe processing system, where each of the horticultural monitorscomprising one or more sensors for monitoring at least one plant andconfigured for recording data from the sensors and transmitting the datato the processing system. The computer-readable medium includesinstructions for causing the at least one processor to perform steps.The steps include obtaining the data for the monitored plant, comparingthe data for the monitored plant with data associated with other plantsof a same species as the monitored plant and at one or more plant lifestages to yield a comparison, and determining, based on the comparisonand via a life stage determination algorithm, a one of the plant lifestages associated with the monitored plant.

The method can include, when the determining indicates that the one ofthe plant life stages for the monitored plant is a target plant lifestage, sending a push notification to a user device, the pushnotification indicating that the plant has reached the target plant lifestage. The method can also include, when the determining indicates thatthe one of the plant life stages for the monitored plant is a targetplant life stage, sending a push notification to a harvester device, thepush notification comprising instructions for the harvester device toharvest at least a portion of the monitored plant.

In the system, the data for the monitored plant can include image data.The steps can then include determining at least one of a species of themonitored plant and the one of the plant life stages for the monitoredplant based on the image data.

When the monitored plant is a fruit bearing plant, the steps can includeestimating, based on the data for the monitored plant, a quantity of atype of fruit that is available for harvesting from the plant.

The steps can include transmitting, over a wireless interface, the datafor the monitored plant from a horticultural monitor to a local beacon,and transmitting the data for the monitored plant from the local beaconto cloud resources via a wide area network, wherein the comparing isperformed by the cloud resources.

When at least one of the plant life stages includes a plant life stageassociated with a suboptimal health condition, the steps can theninclude, when the determining indicates that the one of the plant lifestages for the monitored plant is associated with the suboptimal healthcondition, sending a push notification to a user device, the pushnotification indicating that a health of the monitored plant issuboptimal.

The steps can include determining, using the data and a healthdetermination algorithm, a cause for the health of the monitor plant tobe suboptimal. The steps can then include, based on the cause, providingone or more instructions to the user device for enhancing the health ofthe monitored plant. The steps can also include, based on the cause,determining a remedial measure for enhancing the health of the monitoredplant that does not require human intervention and implementing theremedial measure. The remedial measure can include activating at leastone of an irrigation system or a nutrient delivery system.

In the system, the sensors include at least one of a soil sensor, acamera, or a water sensor.

In the system, the processing system can include a local beacon andremote cloud resources, with the local beacon being communicativelycoupled to one or more horticultural monitors and one or more localresources, and with the remote cloud resources being communicativelycoupled to the local beacon, wherein the comparing and the determiningis performed by the remote cloud resources.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-recited and other advantages and features of the disclosurewill become apparent by reference to specific embodiments thereof whichare illustrated in the appended drawings. Understanding that thesedrawings depict only example embodiments of the disclosure and are nottherefore to be considered to be limiting of its scope, the principlesherein are described and explained with additional specificity anddetail through the use of the accompanying drawings in which:

FIG. 1 shows an example system configuration according to someembodiments;

FIG. 2 shows a network topology according to some embodiments;

FIG. 3 shows an example garden layout according to some embodiments;

FIG. 4 shows an example interface on an interaction device according tosome embodiments;

FIG. 5 shows an example method according to some embodiments;

FIG. 6 shows an example method according to some embodiments;

FIG. 7 shows an example method according to some embodiments;

FIGS. 8A-8F show example interfaces on personal electronic devicesaccording to various embodiments;

FIG. 9 shows an example method according to some embodiments;

FIG. 10A shows an example possible system embodiment for implementingvarious embodiments of the present technology; and

FIG. 10B shows an example possible system embodiment for implementingvarious embodiments of the present technology.

DETAILED DESCRIPTION

Various embodiments of the disclosure are discussed in detail below.While specific implementations are discussed, it should be understoodthat this is done for illustration purposes only. A person skilled inthe relevant art will recognize that other components and configurationsmay be used without parting from the spirit and scope of the disclosure.

The disclosed technology addresses the need in the art for a plantmonitoring system.

FIG. 1 shows an example system configuration according to variousembodiments. Horticultural monitor 101 can monitor one or more plants110 _(a)-110 _(d) (collectively, “plants 110”) using one or more sensors102 _(a)-102 _(e). Horticultural monitor 101 can be a portableelectronic device (e.g., a phone, tablet, or laptop), a general purposecomputing device (e.g., a personal computer), or a special purposedevice (e.g., a device that is purpose-built as horticultural monitor101). Horticultural monitor 101 can have a processor, memory, one ormore communication interfaces (e.g., for wireless or wiredcommunications or any protocol known in the art), a power source,receivers for sensor wires, internal sensors (e.g., one or more ofsensors 102 _(a)-102 _(e)), a power supply for sensors, etc.Horticultural monitor 101 can be a low-power type such that it does notrequire an external power source or recharging. This can be accomplishedby having an internal battery that lasts greater than a year or anenvironmental power harvesting module (e.g., a module for harvestingsolar energy, wind energy, thermal energy, or other sources ofenvironmental energy). Horticultural monitor 101 can have a physicalrestraint mechanism. For example, horticultural monitor 101 can have astake for restraint in soil. Horticultural monitor 101 can have variousother input and output capabilities. For example, horticultural monitor101 can have a keyboard, mouse, motion sensor, microphone, camera,capacitive touch sensor, etc. Horticultural monitor 101 can have adisplay (e.g., LCD, LED, or electronic ink), light (e.g., LED orincandescent bulb), speaker, and/or other output device and/orconnection to receive an output device. Horticultural monitor 101 cancommunicate (e.g., wirelessly or via a wire) with the sensors.Horticultural monitor can also communicate with other devices, whethervia a direct connection (e.g., infrared, ad-hoc WiFi, Bluetooth, etc.)or via a network (e.g., a LAN or WAN such as the Internet).Horticultural monitor 101 can be weather resistant (e.g., against wateror dust) and have a strong outer casing to prevent other environmentaldamage (e.g., from an animal).

In some embodiments, sensors are cameras. For example, sensor 102, isshown as being a camera that is positioned above the plants 110 _(a)-110_(e). The camera can be effective to detect various visible and/orinvisible light spectra (e.g., infrared, red-green-blue, red, green,blue, x-ray, etc.). As such, the camera 102, can be an infrared cameraor a specialized camera which has the capability of detecting individualprogrammed spectra. The camera 102 _(e) can be dynamically positioned byhorticultural monitor 101. For example, horticultural monitor 101 canposition sensor 102, to observe plant 110 _(a) and then position sensor102, to observe plant 102 _(b). The camera 102, can enable variousphysical filters to modify the image. For example, horticultural monitor101 can instruct the camera 102, to enable a red and blue filter to onlyallow green light to be recorded. In another example, horticulturalmonitor 101 can instruct the camera to enable a filter to only letinfrared light pass. Polarizer filters can also be used for variouspurposes (e.g., to decrease glare or identify reflective elements suchas moisture). The camera can detect depth data. Digital filters (e.g.,low pass, high pass, de-noise, dust detection/removal, chromaticaberration correction, lens distortion correction, edge detection, andvarious color and tonal filters) can also be applied in-camera or in aseparate system. The camera can produce various images of the same scenewhich can be analyzed to identify characteristics of a subject plant.

A sensor can be a capacitive soil moisture sensor. For example, thesensor can be a printed circuit board (PCB) with separated traces thatchange capacitance based on the water concentration surrounding thetraces. A sensor as used herein can refer to at least one of the one ormore sensors 102 _(a)-102 _(e). A sensor can be a humidity sensor. Asensor can be an ambient temperature sensor. A sensor can be a topicaltemperature sensor (e.g., to measure the soil temperature). A sensor canbe a luminosity sensor (e.g., a photovoltaic sensor). A sensor can be anelectrical conductivity sensor.

A sensor can be assigned one plant or part of a plant (e.g., on anindividual leaf, branch, fruit, or the stem). A sensor can be assignedmultiple plants or a region. For example, in a garden of 16 plants (4rows by 4 columns), 4 sensors dispersed throughout the garden canmonitor various regions. The environmental characteristics of anindividual plant can be extrapolated from regional sensors. For exampleby using a distance-weighted average of surrounding sensors.

A sensor can provide raw (i.e., analog) values to horticultural monitor101. A sensor can communicate to horticultural monitor 101 using digitalsignals. The communication to horticultural monitor 101 can be wireless(e.g., infrared, cellular, WiFi, other electromagnetic spectra, NFC, ormagnetic fields) or wired. Any wireless protocol can be utilized forcommunication. Sensors can connect to horticultural monitor 101 inparallel, using a bus, or any other means as known in the art. In someembodiments, horticultural monitor 101 supplies power to one or moresensors.

Sensor processing module 104 can use the input values from sensors togenerate data. In some embodiments this includes digitizing raw values(e.g., using an analog-to-digital converter) or otherwise converting thesignal from sensors to more manageable values. Sensor processing module104 can interpolate sensor values and assign them to individual plants.For example, sensor processing module can generate reports on individualplants 110 based on the corresponding sensors.

Communications module 103 can facilitate communication with externaldevices according to some embodiments. The module 103 can be a wirelessmodel that communication with the external devices through any wirelessprotocol.

Horticultural monitor 101 can use ambient power sources (e.g., solar orwind) also for its power. Horticultural monitor 101 can generate powerfrom a local irrigation system. Horticultural monitor 101 can have aninternal battery. Horticultural monitor 10 can utilize high efficiencycomponents and protocols in order to prolong its operating life on asingle power cycle. For example, horticultural monitor 101 can putitself (e.g., the processor, memory, and other components) in a suspendstate with a wakeup time. Thus, horticultural monitor 101 canperiodically wake up to take measurements and transmit data followed byanother suspend cycle.

FIG. 2 shows a network topology according to some embodiments.Horticultural monitors 101 _(a), 101 _(b), 101 _(c), . . . , 101 _(n)can communicate with local beacon 201. Local beacon 201, Interaction(user) devices 203 _(a), 203 _(b), 203 _(c), . . . , 203 _(n)(collectively, “interaction device 203”), cloud resources 204, and thirdparty APIs 205 _(a), 205 _(b), 205 _(c), . . . , 205 _(n) (collectively,“third party API 205”) can interconnect via the Internet 202 or anyother network.

Local beacon 201 can connect to horticultural monitor 101 via a wirelessprotocol. Because of possible power constraints, the communicationbetween local beacon 201 and horticultural monitor 101 can utilizeefficient protocols such as Bluetooth low energy and Zigbee. However,higher power protocols are also contemplated and can be sufficient.Local beacon 201 can be connected to a user's home router using WiFi,Ethernet, or other Internet protocols. Local beacon 201 can preprocessdata from multiple horticultural monitors 101. Local beacon 201 can sendhorticultural monitor 101 data to interaction device 203, cloudresources 204, or third party APIs 205.

For example, a sensor can collect sensor data as described herein. Thissensor data can describe one or more of the humidity, temperature, windspeed, air quality, soil composition, soil water saturation, soilconductivity, soil pH, soil water outflow, ambient luminosity, plantcharacteristics (height, temperature, conductivity, chlorophyll levels,etc.), pest detections, etc. The sensor data can also include cameradata (e.g., visible, infrared data, or data associated with otherspectra). The sensor data can also include status data regarding thestatus and identity of the individual sensors (e.g., power status,location, and serial number). Horticultural monitor 101 can then receivethe sensor data and transmit the sensor data to local beacon 201. Beforerelaying the sensor data, horticultural monitor 101 can interpret orotherwise process some or all of the sensor data; this can includesummarizing the sensor data. Local beacon 201 can then receive thesensor data and similarly interpret or otherwise process some or all ofthe received data. Local beacon 201 can then send the data tointeraction device 203, cloud resources 204, or third party APIs 205.

Local beacon 201 can host a web server or application program interface(“API”) so that external devices and users can read and interpret dataon local beacon 201. Interaction device 203 can connect to local beacon201 either directly or via a home network, bypassing the Internet 202.This can be advantageous according to security, bandwidth, andreliability priorities.

Cloud resources can process data from local beacon 201 and present thedata to interaction device 203. For example, a user can—usinginteraction device 203—logon to a website that is hosted by cloudresources 204 to view data regarding their plants 110. Cloud resourcescan facilitate the configuration of horticultural monitor 101 (includingone or more sensor) and local beacon 201 as disclosed herein. Cloudresources 204 can push updates to horticultural monitor 101.

Third party APIs 205 can provide additional interaction or data. Forexample, a weather service can provide one or more of rainfall,sunlight, and cloud-cover data instead of or in addition to sensor data.Third party APIs can also be used to publish information; for example,cloud resources can publish a photo from camera 102, and plant healthstatistics to a social media account such as Pinterest, Facebook,Twitter, Instagram and so forth.

The plant monitoring system can include any combination of thehorticultural monitor 101, the local beacon 201, the Internet 202 (i.e.,a wide area network), the interface device 203, the cloud resources 204,and the third party APIs 205. Many of the features herein disclosed canbe performed by one or multiple of the entities shown in FIG. 2.

FIG. 3 shows an example garden layout 300 according to some embodiments.In example garden layout 300 various plants, sensors, and horticulturalmonitors can be placed in various locations. A sensor can be assignedand placed with an individual plant (e.g., at B2, the sensor is assignedto one plant) and in some embodiments a sensor can be placed such thatit covers a region (e.g., plants at B4, D4, F4, B6, D6, and F6 can beassigned to have their data gathered from the sensors at C5 and E5).Multiple horticultural monitors can service a garden layout 300. Forexample, the horticultural monitor at C6 can service the sensors at C5and E5 while the horticultural monitor at F2 can service the sensors atB2, C2, D2, C3, and F4. Multiple horticultural monitors can service thesame sensor for redundancy and accuracy purposes.

FIG. 4 shows example interface 402 on an interaction or mobile device203. The interface 402 can be part of an application that is operable ona mobile device 203. Interface 402 can display various plants associatedwith the user. For example, it can show a picture 404 of the plant,portion of the plant, or a graphic such as a stock image to representthe plant. Relevant information 401 can also be shown. For example, inFIG. 4, relevant information 401 includes the moisture content of thesoil, which in this example is 70%. Relevant information 401 can includecurrent values and comparison with target values. For example, one plantmay require four hours of sunlight per day and relevant information candepict the amount of sunlight the plant has currently received for theday, thus giving the viewer an idea of how much more that plant needs.The data can also include remaining daylight and expected sunlight forthe remainder of the day, week or other time frame. Relevant information401 can show a warning of a certain characteristic is outside of acertain range. For example, if the pH of the soil is greater or lessthan an optimal range for the specific plant. Relevant information 401can provide the age of the plant and the time until maturity. Forexample, if the plant produces fruit, relevant information 401 canindicate how long until the fruit will be ready to harvest. A user canregister a new plant to their monitored list by clicking add button 403.

A user can activate and control various parts of the system using aninterface (e.g., example interface 402). For example, the user cancontrol the camera (e.g., activate filters, move the camera, or take apicture or video), the user can instruct the horticultural monitor toperform tasks such as deliver nutrients (e.g., fertilizer, water, orcompost), activate shades, or activate pest deterrent measures (e.g., anaudio buzzer, pesticides, or a laser). Using the interface, a user canqueue and schedule these activities. Similarly, the user can setconditional rules to activate these activities (e.g., when watersaturation is a certain percentage, turn off irrigation). In thisregard, various triggers can be established which, when one occurs,causes an action to be taken in the system.

A plant can be registered by a user scanning the plant (e.g., using abar-code on the plant container or seed bag), taking a picture of theplant and using pattern recognition software to determine its species,or manually entering its information using an interface device 203. Forexample, using interface 402, a user can select add button 403 andinterface 402 can present various plant species for selection. A usercan select a picture of the plant; alternatively, a user can search forthe plant using text. A user can describe the plant (e.g., how manyleaves, the shape of the leaves, how the stem branches, or the size ofthe plant) and the interface 402 can suggest plants that fit thedescription.

In continuing the registration process, a user can associate sensors tothe new plant. For example, a user can scan a sensor (e.g., a barcode onthe sensor) or input a unique ID from the sensor. The system canautomatically register a sensor (but not yet associate it with a plant)when the sensor is within range or when connected to a horticulturalmonitor. A user can then associate the sensor with the plant or indicatea location in a garden layout. The user can indicate a desired pollingfrequency of the sensor. For example, if the sensor is a camera, a usercan indicate that the camera should take a new picture every hour orevery day.

In continuing the registration process, a user can indicate the locationof the new plant. For example, interface 402 can show a garden layoutsimilar to example garden layout 300. The user can indicate where in thegarden layout (e.g., grid location E3) that the plant is located. Thesystem can automatically determine the location of the plant using, forexample, a camera or any location based device. The user can indicatecharacteristics about the location of the plant. For example, the usercan indicate that the plant is “inside, by the window.” This can helpwith identification of the plant. A user can name the plant.

FIG. 5 shows an example method according to some embodiments.Horticultural monitor can begin by recording a plant image (step 502)for example, by using a camera attached to the horticultural monitor.The horticultural monitor can then compress the image (step 504). Thiscan include taking the raw data from the camera and converting it into asmaller size by decreasing its resolution. Compression algorithms suchas JPEG can also be used to make the image a smaller size. In someembodiments, step 504 is performed by the camera itself. A systemperforming the method can continue by having the horticultural monitortransmit the image to the local beacon (step 506). The horticulturalmonitor can transmit the image immediately after it is captured;alternatively, the horticultural monitor can wait until a determinedtime before transmitting the image. For example, the horticulturalmonitor can hold images until a certain time when it can subsequently“dump” the images to the local beacon. The horticultural monitor (via acamera) can constantly be capturing images; alternatively, images can becaptured only on command from local beacon (or other part of the plantmonitoring system). The local beacon can then receive the image (step508).

A system performing the method can continue by having the imageprocessed (step 510). This can include correcting optical imperfectionsin the image (e.g., color aberration, lens distortions, dead pixels,etc.). Processing the image can further include identifying features inthe image (as disclosed herein). A system performing the method can theninclude appending the image to a time lapse video (step 512).

FIG. 6 shows an example method according to some embodiments. Thismethod can be practiced by a computing device or a computer thatdetermines the plant species (step 602). In some embodiments, step 602includes using a neural network and machine learning to associate animage of a plant with its species. Step 602 can include determining aconfidence index for various plant species. For example it can calculatea 30% confidence index that the image represents one species and a 10%confidence index that the image represents another species.

Once an identification of the plant is achieved, the device or computercan continue by having the system determine the plant life stage (step604). For example, the system can determine when the plant has sprouted,when it has reached maturity, when it has developed leaves, fruit,flowers, etc. Step 604 can use neural networks and machine learning todetermine the plant life stage and assign confidence indices as such.The system can then determine if the plant is entering a new life stage(step 606), if it is then it can push a notification to the user (608).For example, a new life stage can include when the plant begins to growfruit or its flowers begin to blossom. Step 608 can include sending apush notification to a user's personal interaction device (e.g.,interaction device 203), based on the new life stage that is detected.Step 608 can include using a third party API 205 to publish a post on athird-party platform such as social media such as, for example,Facebook, Instagram, Pinterest, or Twitter.

FIG. 7 shows an example method according to some embodiments. The methodinvolves a hardware component such as a computer or monitoring deviceactivating a filter on the camera (step 702). The filter can be set forany spectra or color. Step 702 can include activating a motor that movesthe filter physically over the camera lens or sensor. The camera canthen capture an image (step 704). For example, if the filter is a redfilter, the image captured would be a near-infrared image. In someembodiments, the camera includes a sensor that records red, green, andblue channels (“RGB camera”). With a filter blocking out most of thechannel according to the filter's structure, the unfiltered light thatis detected can be useful for various plant identification anddiagnostic processes. A blue filter can be utilized to isolate anear-infrared image. The camera can be an infrared camera and cancapture an infrared image. A system performing the method can continueby capturing a color (e.g., RGB) image (step 706). If a filter iscovering the lens, the filter can be moved. Other physical filters canbe applied such as a polarizing filter to enhance or diminish certainoptical characteristics of an image.

A system performing the method can continue at step 708 by isolating thegreen channel of the color image to detect leaves. In some embodiments,this includes discarding the red and blue channel information. A systemperforming the method can continue and mask out non-green parts of thecolor image (step 710). For example, step 710 can include masking outwhere the green channel is below a certain threshold and/or where thehue of a pixel is not within a green range. In some embodiments, step710 includes the computer applying pixel clustering and blur techniquesso as to discard outlier pixels (e.g., those resulting from colornoise).

A system performing the method can continue at step 712 by calculatingthe normalized difference vegetation index (NDVI) for each pixel. Thisindex can be calculated using the formula NDVI=(R−B)/(R+B)+1 where “R”is red value of the pixel and “B” is blue value of the pixel. A systemperforming the method can then superimpose a photosynthesis colormap onthe color image (step 714). The photosynthesis colormap can be createdusing the NDVI. A system performing the method can then calculate anenhanced normalized difference vegetation index (step 716). This can bedone using the formula ((R+G)−2*B)/((R+G)+2*B)+1.

A system performing the method can then continue and identify insectsand pests (step 718). For example, the system can identify actual pests(such as caterpillars) or the signs of pests (such as leaves beingeaten). Step 718 can include identifying other organisms such as fungusor bacteria that might be on the plant, in the plant, or near the plant(e.g., in the soil, air, or neighboring structure). Step 718 can useneural networks and machine learning.

Similar to step 604, step 720 of the method can include identifying theripeness of the plant. For example, it can make such determination basedon the color or size of a fruit. Other factors may be used as well todetermine ripeness, such as odor or time of the season. Step 720 caninclude comparing an image of the plant or portion of the plant with adatabase of images. For example, if a captured image is similar to adatabase image or collection of database images that are associated withan identified ripeness, the system performing step 720 can determinethat the plant has the identified ripeness. This similarity can bedetermined based on a variety of qualities of the plant or part of theplant in the image. Examples of such a variety of qualities includesize, texture, color, physical temperature, transparency, weight,reflectiveness, density of features (e.g., leaves, fruits, branches, orseeds), etc. The system can determine a confidence level for thedetermination of the ripeness. The system can determine a percentageripeness if, e.g., part of the plant is ripe while other parts are notyet ripe. Step 720 can utilize computer vision (e.g., edge detection,feature identification, feature measuring—including color analysis, andobject recognition) and machine learning to identify ripeness. Machinelearning can include using a dataset of example images to determine thequalities (e.g., measurements) characteristic of defined labels assignedto the example images (e.g., “ripe” or “not ripe”). Such qualities canbe utilized for the computer vision analysis.

Combining features herein disclosed can assist in the care andmonitoring of plants in a personal or commercial setting. The plantmonitoring system can detect the ripeness of a fruit and send an alertto a user with a photo of the plant and/or fruit. This can remind aless-attentive user to check on the fruit (e.g., if they are busy or outof town). The system can detect new plant life stages as hereindisclosed and notify a user as herein disclosed. The system can utilizemachine learning to recognize the plant species.

The plant monitoring system can educate a user about the care of theirplant. For example, the system can indicate the optimal conditions(e.g., temperature, sunlight, humidity, water, etc.) for their plant andprovide suggestions for achieving these optimal values. For example, itcould provide tips for how to keep a plant warm during the winter or atnight. The plant monitoring system can, using data from sensors andother sources, determine the types of plants that would flourish in thegiven environment. Videos and tutorials that are specific to a plantspecies or general to a genus or all gardening can be available via theinterface device. The system can identify that the plant is sick orhealthy and instruct the user on how to identify such symptoms (e.g.,“note that the leaves have a texture like this image, this indicatesthat it may have ______ disease”).

The sensors can include a depth sensor or depth sensor to providespecial information. This spatial data in combination with pictures andother visual data can help create a three-dimensional model of theplant. The three-dimensional model can then be used for entertainment aswell as technical purposes. For example, the three-dimensional model canbe compared with structural data of similar plants to determine if it ishealthy or within normal size expectations. The plant monitoring systemcan suggest that a user trim or prune certain parts of the plant. Thiscan benefit from using virtual or augmented reality technology and thethree-dimensional model. The system can save three-dimensional snapshotsof the plant over time in order to create a time lapse of its growth anddevelopment.

The plant monitoring system can help a user to assess the needs of thesoil for their plants. For example, using the sensors, it may determinethat the pH, assay nitrogen, phosphorus, potassium, calcium, sulfur,magnesium, etc., levels are not optimal. The system can providesuggestions for fixing any deficiencies or problems. For example, it cansuggest that the user place a type of compost or fertilizer in the soil.The soil can be monitored to determine if a plant infection has spreadto the soil.

System can be used to count plants, leaves, fruits, amount of water pergiven period of time, etc. The system can also determine the volume ofthe fruits etc. These accountings can help a user can predict a harvest.In some embodiments, this is done by calculating the area or volume ofplants or fruits and determining a total count based on average plant orfruit density.

An infrared sensor can determine the overall health of a plant. If thesystem detects poor health it can notify a user. The system can useenvironmental data (e.g., weather, sunlight, etc.) as well as sensordata to estimate the plant health and create a health index. The healthindex can be a value that describes the health of the plant. The systemcan warn a user of a future event or present condition (e.g., hot day,day of full sun, cold snap, high humidity, rain, migration of insects,etc.) and advise the user of precautionary measures to protect theplant. Data for future events can come from weather forecasts.

Chlorophyll can be measured over time using an infrared camera and timeswhere the chlorophyll drops can be identified. An analysis of thechlorophyll levels and chlorophyll distribution in the leaves of a plantcan be useful in disease detection. An image of one leaf can be comparedwith a historical image, an image of another leaf on the same plant, ora database of similar plants to determine if the leaf shows signs ofdisease. In order to isolate the chlorophyll emissions of weak nearinfrared light, a light (e.g., a blue light) can be shown on the leaf atnight; this can prevent other ambient radiation from adversely affectingthe image.

In some embodiments, the system includes a sensor that monitors the flowof water from the soil. For example, tubes underneath a garden canmeasure a sampling of water flow. This can be used to determine theoverall irrigation loss volume. Irrigation loss volume combined withirrigation meters, rain meters, and other sensors can inform how muchwater has evaporated, how much is retained within the soil, and how muchwater has been used by the plant.

In some embodiments, the plant monitoring system not only monitors theplant but can actively participate in its development. For example,local resources can be connected to the horticultural monitor or localbeacon to perform various functions, such as irrigation, shading, noise,heating, cooling, or nutrient systems. For example, the system cantrigger irrigation to a plant or group of plants. The system can alsoemit a noise to scare off pests (e.g., deer or squirrels). A pet canwear a device that is easy to detect by the system so that when the petis close to a plant, the system can emit a warning signal to scare thepet away. The system can engage a shade to block or obscure the sunlightgetting to the plant. The system can activate a heater to keep the plantat an optimal temperature. The system can deliver fertilizer to thesoil. The system can attempt to eliminate pests (e.g., using lasers orfocused light). The system can activate a dehumidifier or humidifier tocontrol humidity around the plant. The system can use a dehumidifier tocapture water when the plant's stomata are closed and use that water forirrigation. The system can reflect sunlight to a plant. This can beespecially useful if the plant is in a shaded, partially shaded, orseasonally shaded area.

The system can facilitate the breaking down of compost below the plantsfor heat generation. The system can then monitor and control theexothermic reactions.

The system disclosed herein can be utilized on fruits, vegetables, andflowers that have been extracted from the soil (e.g., within a grocerystore). The system can monitor the moisture, ripeness, and freshness ofthese items and inform a worker to address any problems or removeinventory.

The plant monitoring system can be used to monitor plants etc. inunderwater environments, such as within a fish tank or lake. Instead ofmeasuring soil quality it can measure the water quality.

FIG. 8A shows an example interface on a personal electronic device(e.g., interaction device 203) according to various embodiments. Forexample, FIG. 8A shows various connected plants in an interface. Theconnected plants that need attention (e.g., more watering, sunlight, orother care) can be identified in a list of plants. The update frequencyof each or all connected plants can be changed.

FIG. 8B shows an example interface on a personal electronic device(e.g., interaction device 203) according to various embodiments. Forexample, FIG. 8B shows an example connected plant display. Afterreceiving a selection of a connected plant in FIG. 8A, the device canshow the respective display of FIG. 8B). This display can show at leastphotos of the plant (including the option to take a photo), warnings forthe plant (e.g., that the plant needs more sun or water), an option toremedy a problem identified in the warning (by, e.g., watering theplant), upcoming events (e.g., weather events or plant care events), anda history (including notes) of the plant.

FIG. 8C shows an example interface on a personal electronic device(e.g., interaction device 203) according to various embodiments. Forexample, FIG. 8C shows an example technique for adding a plant as aconnected plant.

FIG. 8D shows an example interface on a personal electronic device(e.g., interaction device 203) according to various embodiments. Forexample, the device can show FIG. 8D after receiving a selection of anew plant as shown in FIG. 8C. The system can display and receiveselections for a plant species, a custom name or avatar, a description,a life stage (e.g., seedling or sapling), a connected sensor (e.g., asensor that monitors the plant), a date planted, etc.

FIG. 8E shows an example interface on a personal electronic device(e.g., interaction device 203) according to various embodiments. Forexample, FIG. 8E shows an example plant hydration interface. The planthydration interface can indicate the amount of water the plant hasreceived or the soil saturation level. The plant hydration interface canindicate the last time the plant received water (e.g., through rain orirrigation). The plant hydration interface can receive input forwatering the plant.

FIG. 8F shows an example interface on a personal electronic device(e.g., interaction device 203) according to various embodiments. Forexample, FIG. 8F shows an example plant hydration interface wherein theplant needs water.

FIG. 9 shows an example method that can be performed by system(s)disclosed herein. Such a system can begin and capture an image of aplant of a certain species (step 902). Step 902 can be accomplished witha sensor (e.g., a camera) as disclosed herein.

The system can continue performing the example method of FIG. 9 bytransmitting, over a wireless interface, the image of the plant from aplant monitor to a local beacon (step 904). Step 904 can be an exampleof step 506 as discussed above.

The system can continue performing the example method of FIG. 9 bytransmitting the image of the plant from the local beacon to cloudresources via a wide area network (step 906). Step 906 can be an exampleof step 508 as discussed above. The Internet is an example of a widearea network.

The system can continue performing the example method of FIG. 9 bydetermining the certain species of the plant based on the captured image(step 908). Step 908 can be an example of step 602 as discussed above.The system can identify the species in any number of ways. For example,the image can be compared graphically to images of known species todetermine how similar the plants look to a known species. An algorithmcan also evaluate the image for certain colors, shapes, and/or patternsto determine the species as well, without any direct reference to apicture of the known species. The algorithm can include data such asknown temperatures, time of year, geographic location, and so forth,which data can affect the condition of the species. These and othercontemplated approach can be used to determine the species of the plant.

The system can continue performing the example method of FIG. 9 byestimating a quantity of a type of fruit that is available forharvesting from the plant (step 910). This approach can of courseinclude aggregating data from a number of different plants to provide anaggregated estimate of the quantity of fruit that is available forharvesting. The system can also present a graphical approach to notifyindividuals of which plants have fruit ready to harvest.

The system can continue performing the example method of FIG. 9 bycomparing, via a processor, the image of the plant with images of ripeplants of the certain species to yield a comparison (step 912). Step 912can be an example of step 720. This can be done in many different wayssuch as comparing an image to an image of a ripe plant, analyzing one ormore of the color, texture, shape, weight, bend or condition of theplant (how bent over is a stalk holding the fruit), and so forth. Date,location, temperature, humidity, weather conditions, and/or otherfactors can be also applied to determine ripeness.

The system can continue performing the example method of FIG. 9 bydetermining, based on the comparison and via a determination algorithm,whether a plant is ripe (step 914). Step 914 can be an example of step720. If the system determines in step 914 that the plant is ripe, it cancontinue and send a push notification to a user device, the pushnotification including the image of the plant and indicating that theplant is ripe (step 916). In some embodiments, the push notificationdoes not include an image. The push notification can be similar thatwhich is described in step 608. The push notification can include a link(e.g., a URL) to a website, application, etc. describing the plant. Thepush notification can include text identifying the plant. The pushnotification image can be formatted for display on a variety of devices(e.g., watch, phone, laptop or tablet). The push notification caninclude a selectable option (e.g., to water the plant, turn on a lamp,etc.). The push notification can provide a user with the option ofselling the plant or fruit of the plant. The image can include anindication of ripeness (e.g., highlighting fruit that is ripe or a greencheck mark).

The push notification can be effective to indicate that the plant isripe. For example, an indicator (e.g., a colored dot, text, a briefmulti-media notification) can be presented in association with the plantto indicate its ripeness. Such an indicator can be a vibration, agraphic, a sound, etc. The indicator can be presented within a page forthe associated plant or elsewhere (e.g., in a notification bar, dock,task bar, etc. of an operating system).

FIG. 10A and FIG. 10B show example possible system embodiments. The moreappropriate embodiment will be apparent to those of ordinary skill inthe art when practicing the present technology. Persons of ordinaryskill in the art will also readily appreciate that other systemembodiments are possible.

FIG. 10A illustrates a conventional system bus computing systemarchitecture 1000 wherein the components of the system are in electricalcommunication with each other using a bus 1005. Example system 1000includes a processing unit (CPU or processor) 1010 and a system bus 1005that couples various system components including the system memory 1015,such as read only memory (ROM) 1020 and random access memory (RAM) 1025,to the processor 1010. The system 1000 can include a cache of high-speedmemory connected directly with, in close proximity to, or integrated aspart of the processor 1010. The system 1000 can copy data from thememory 1015 and/or the storage device 1030 to the cache 1012 for quickaccess by the processor 1010. In this way, the cache can provide aperformance boost that avoids processor 1010 delays while waiting fordata. These and other modules can control or be configured to controlthe processor 1010 to perform various actions. Other system memory 1015may be available for use as well. The memory 1015 can include multipledifferent types of memory with different performance characteristics.The processor 1010 can include any general purpose processor and ahardware module or software module, such as module 1 1032, module 21034, and module 3 1036 stored in storage device 1030, configured tocontrol the processor 1010 as well as a special-purpose processor wheresoftware instructions are incorporated into the actual processor design.The processor 1010 may essentially be a completely self-containedcomputing system, containing multiple cores or processors, a bus, memorycontroller, cache, etc. A multi-core processor may be symmetric orasymmetric.

To enable user interaction with the computing device 1000, an inputdevice 1045 can represent any number of input mechanisms, such as amicrophone for speech, a touch-sensitive screen for gesture or graphicalinput, keyboard, mouse, motion input, speech and so forth. An outputdevice 1035 can also be one or more of a number of output mechanismsknown to those of skill in the art. In some instances, multimodalsystems can enable a user to provide multiple types of input tocommunicate with the computing device 1000. The communications interface1040 can generally govern and manage the user input and system output.There is no restriction on operating on any particular hardwarearrangement and therefore the basic features here may easily besubstituted for improved hardware or firmware arrangements as they aredeveloped.

Storage device 1030 is a non-volatile memory and can be a hard disk orother types of computer readable media which can store data that areaccessible by a computer, such as magnetic cassettes, flash memorycards, solid state memory devices, digital versatile disks, cartridges,random access memories (RAMs) 1025, read only memory (ROM) 1020, andhybrids thereof.

The storage device 1030 can include software modules 1032, 1034, 1036for controlling the processor 1010. Other hardware or software modulesare contemplated. The storage device 1030 can be connected to the systembus 1005. In one aspect, a hardware module that performs a particularfunction can include the software component stored in acomputer-readable medium in connection with the necessary hardwarecomponents, such as the processor 1010, bus 1005, display 1035, and soforth, to carry out the function.

FIG. 10B illustrates a computer system 1050 having a chipsetarchitecture that can be used in executing the described method andgenerating and displaying a graphical user interface (GUI). Computersystem 1050 is an example of computer hardware, software, and firmwarethat can be used to implement the disclosed technology. System 1050 caninclude a processor 1055, representative of any number of physicallyand/or logically distinct resources capable of executing software,firmware, and hardware configured to perform identified computations.Processor 1055 can communicate with a chipset 1060 that can controlinput to and output from processor 1055. In this example, chipset 1060outputs information to output 1065, such as a display, and can read andwrite information to storage device 1070, which can include magneticmedia, and solid state media, for example. Chipset 1060 can also readdata from and write data to RAM 1075. A bridge 1080 for interfacing witha variety of user interface components 1085 can be provided forinterfacing with chipset 1060. Such user interface components 1085 caninclude a keyboard, a microphone, touch detection and processingcircuitry, a pointing device, such as a mouse, and so on. In general,inputs to system 1050 can come from any of a variety of sources, machinegenerated and/or human generated.

Chipset 1060 can also interface with one or more communicationinterfaces 1090 that can have different physical interfaces. Suchcommunication interfaces can include interfaces for wired and wirelesslocal area networks, for broadband wireless networks, as well aspersonal area networks. Some applications of the methods for generating,displaying, and using the GUI disclosed herein can include receivingordered datasets over the physical interface or be generated by themachine itself by processor 1055 analyzing data stored in storage 1070or 1075. Further, the machine can receive inputs from a user via userinterface components 1085 and execute appropriate functions, such asbrowsing functions by interpreting these inputs using processor 1055.

It can be appreciated that example systems 1000 and 1050 can have morethan one processor 1010 or be part of a group or cluster of computingdevices networked together to provide greater processing capability.

For clarity of explanation, in some instances the present technology maybe presented as including individual functional blocks includingfunctional blocks including devices, device components, steps orroutines in a method embodied in software, or combinations of hardwareand software.

Any of the steps, operations, functions, or processes described hereinmay be performed or implemented by a combination of hardware andsoftware modules, alone or in combination with other devices. In anembodiment, a software module can be software that resides in memory ofa client device and/or one or more servers of a content managementsystem and perform one or more functions when a processor executes thesoftware associated with the module. The memory can be a non-transitorycomputer-readable medium.

In some embodiments the computer-readable storage devices, mediums, andmemories can include a cable or wireless signal containing a bit streamand the like. However, when mentioned, non-transitory computer-readablestorage media expressly exclude media such as energy, carrier signals,electromagnetic waves, and signals per se.

Methods according to the above-described examples can be implementedusing computer-executable instructions that are stored or otherwiseavailable from computer readable media. Such instructions can include,for example, instructions and data which cause or otherwise configure ageneral purpose computer, special purpose computer, or special purposeprocessing device to perform a certain function or group of functions.Portions of computer resources used can be accessible over a network.The computer executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, firmware, orsource code. Examples of computer-readable media that may be used tostore instructions, information used, and/or information created duringmethods according to described examples include magnetic or opticaldisks, flash memory, USB devices provided with non-volatile memory,networked storage devices, and so on.

Devices implementing methods according to these disclosures can includehardware, firmware and/or software, and can take any of a variety ofform factors. Typical examples of such form factors include laptops,smart phones, small form factor personal computers, personal digitalassistants, and so on. Functionality described herein also can beembodied in peripherals or add-in cards. Such functionality can also beimplemented on a circuit board among different chips or differentprocesses executing in a single device, by way of further example.

The instructions, media for conveying such instructions, computingresources for executing them, and other structures for supporting suchcomputing resources are means for providing the functions described inthese disclosures.

Although a variety of examples and other information was used to explainaspects within the scope of the appended claims, no limitation of theclaims should be implied based on particular features or arrangements insuch examples, as one of ordinary skill would be able to use theseexamples to derive a wide variety of implementations. Further andalthough some subject matter may have been described in languagespecific to examples of structural features and/or method steps, it isto be understood that the subject matter defined in the appended claimsis not necessarily limited to these described features or acts. Forexample, such functionality can be distributed differently or performedin components other than those identified herein. Rather, the describedfeatures and steps are disclosed as examples of components of systemsand methods within the scope of the appended claims.

What is claimed is:
 1. A computer-implemented method comprising:capturing data for a monitored plant via one or more sensors; comparing,via a processor, the data for the monitored plant with data associatedwith other plants of a same species as the monitored plant and at one ormore plant life stages to yield a comparison; and determining, based onthe comparison and via a life stage determination algorithm, a one ofthe plant life stages associated with the monitored plant.
 2. Thecomputer-implemented method of claim 1, further comprising: when thedetermining indicates that the one of the plant life stages for themonitored plant is a target plant life stage, sending a pushnotification to a user device, the push notification indicating that theplant has reached the target plant life stage.
 3. Thecomputer-implemented method of claim 1, further comprising: when thedetermining indicates that the one of the plant life stages for themonitored plant is a target plant life stage, sending a pushnotification to a harvester device, the push notification comprisinginstructions for the harvester device to harvest at least a portion ofthe monitored plant.
 4. The computer-implemented method of claim 1,wherein the data for the monitored plant comprises image data, andfurther comprising determining at least one of a species of themonitored plant and the one of the plant life stages for the monitoredplant based on the image data.
 5. The computer-implemented method ofclaim 1, wherein the monitored plant is a fruit bearing plant, themethod further comprising: estimating, based on the data for themonitored plant, a quantity of a type of fruit that is available forharvesting from the plant.
 6. The computer-implemented method of claim1, further comprising: transmitting, over a wireless interface, the datafor the monitored plant from a horticultural monitor to a local beacon;and transmitting the data for the monitored plant from the local beaconto cloud resources via a wide area network, wherein the comparing isperformed by the cloud resources.
 7. The computer-implemented method ofclaim 1, wherein at least one of the plant life stages comprises a plantlife stage associated with a suboptimal health condition.
 8. Thecomputer-implemented method of claim 7, further comprising: when thedetermining indicates that the one of the plant life stages for themonitored plant is associated with the suboptimal health condition,sending a push notification to a user device, the push notificationindicating that a health of the monitored plant is suboptimal.
 9. Thecomputer-implemented method of claim 8, further comprising: determining,using the data and a health determination algorithm, a cause for thehealth of the monitor plant to be suboptimal.
 10. Thecomputer-implemented method of claim 9, further comprising: based on thecause, providing one or more instructions to the user device forenhancing the health of the monitored plant.
 11. Thecomputer-implemented method of claim 9, further comprising: based on thecause, determining a remedial measure for enhancing the health of themonitored plant that does not require human intervention; andimplementing the remedial measure.
 12. The computer-implemented methodof claim 11, wherein the remedial measure comprises activating at leastone of an irrigation system or a nutrient delivery system.
 13. Thecomputer-implemented method of claim 1, wherein the sensors comprise atleast one of a soil sensor, a camera, or a water sensor.
 14. A systemcomprising: a processing system comprising at least one processor and acomputer-readable medium; and one or more horticultural monitorscommunicatively coupled to the processing system, each of thehorticultural monitors comprising one or more sensors for monitoring atleast one plant and configured for recording data from the sensors andtransmitting the data to the processing system, wherein thecomputer-readable medium comprises instructions for causing the at leastone processor to perform steps comprising: obtaining the data for themonitored plant; comparing the data for the monitored plant with dataassociated with other plants of a same species as the monitored plantand at one or more plant life stages to yield a comparison; anddetermining, based on the comparison and via a life stage determinationalgorithm, a one of the plant life stages associated with the monitoredplant.
 15. The system of claim 14, the steps further comprising: whenthe determining indicates that the one of the plant life stages for themonitored plant is a target plant life stage, sending a pushnotification to a user device, the push notification indicating that theplant has reached the target plant life stage.
 16. The system of claim14, the steps further comprising: when the determining indicates thatthe one of the plant life stages for the monitored plant is a targetplant life stage, sending a push notification to a harvester device, thepush notification comprising instructions for the harvester device toharvest at least a portion of the monitored plant.
 17. The system ofclaim 14, wherein the data for the monitored plant comprises image data,and the steps further comprising determining at least one of a speciesof the monitored plant and the one of the plant life stages for themonitored plant based on the image data.
 18. The system of claim 14,wherein the monitored plant is a fruit bearing plant, the method furthercomprising: estimating, based on the data for the monitored plant, aquantity of a type of fruit that is available for harvesting from theplant.
 19. The system of claim 14, wherein at least one of the plantlife stages comprises a plant life stage associated with a suboptimalhealth condition.
 20. The system of claim 14, further comprising: whenthe determining indicates that the one of the plant life stages for themonitored plant is associated with the suboptimal health condition,sending a push notification to a user device, the push notificationindicating that a health of the monitored plant is suboptimal.
 21. Thesystem of claim 20, further comprising: determining, using the data anda health determination algorithm, a cause for the health of the monitorplant to be suboptimal.
 22. The system of claim 21, further comprising:based on the cause, providing one or more instructions to the userdevice for enhancing the health of the monitored plant.
 23. The systemof claim 21, further comprising: based on the cause, determining aremedial measure for enhancing the health of the monitored plant thatdoes not require human intervention; and implementing the remedialmeasure.
 24. The system of claim 14, wherein the processing systemcomprises a local beacon and remote cloud resources, the local beaconbeing communicatively coupled to one or more horticultural monitors andone or more local resources, and the remote cloud resources beingcommunicatively coupled to the local beacon, wherein the comparing andthe determining is performed by the remote cloud resources.